Multi-function time-of-flight sensor and method of operating the same

ABSTRACT

A method of operating a time-of-flight (ToF) sensor including at least one depth pixel having a multi-tap structure and a light source illuminating a transmission light to an object is provided. An operation mode of a ToF sensor is determined among a distance detection mode to sense a distance to an object and a plurality of additional operation modes. A plurality of taps of a depth pixel and a light source are controlled based on the determined operation mode such that the plurality of taps generate a plurality of sample data corresponding to the determined operation mode. A sensing result corresponding to the selected operation mode is determined based on the plurality of sample data. A plurality of functions, in addition to a function of the ToF sensor to measure a distance to an object, may be performed efficiently by controlling the plurality of taps of the depth pixel and the light source depending on the operation modes.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority under 35 USC § 119 from Korean PatentApplication No. 10-2020-0088776, filed on Jul. 17, 2020, in the KoreanIntellectual Property Office (KIPO), the disclosure of which isincorporated by reference herein in its entirety.

BACKGROUND 1. Technical Field

Example embodiments relate generally to semiconductor integratedcircuits, and more particularly to a multi-function time-of-flight (ToF)sensor and a method of operating a ToF sensor.

2. Discussion of the Related Art

Recently, interest in sensing technologies for acquiringthree-dimensional information of an object is increasing and variousthree-dimensional cameras have been developed. Among thethree-dimensional cameras, a ToF (Time-of-Flight) sensor has a simplecircuit configuration and high distance resolution. The ToF sensorilluminates an object with a transmission light using a light source,and calculates a distance to the object by measuring a phase differenceof a flight time of a reception light reflected from the object usingdemodulation signals. Various electronic devices, including mobilephones, may include sensors having different functions, in addition tothe ToF sensor, thereby increasing the sizes of the electronic devices.

SUMMARY

Some example embodiments may provide a time-of-flight (ToF) sensorcapable of various functions, and a method of operating the ToF sensorto implement various functions.

According to an aspect of an example embodiment, a method of operating atime-of-flight (ToF) sensor including at least one depth pixel having amulti-tap structure and a light source illuminating a transmission lightto an object, includes, determining an operation mode of a ToF sensor,from among a distance detection mode to sense a distance to an objectand a plurality of additional operation modes, controlling a pluralityof taps of a depth pixel and a light source based on the determinedoperation mode such that the plurality of taps generate a plurality ofsample data corresponding to the determined operation mode, anddetermining a sensing result corresponding to the determined operationmode based on the plurality of sample data.

According to an aspect of another example embodiment, a time-of-flightsensor includes a light source configured to illuminate a transmissionlight to an object, a pixel array comprising at least one depth pixelhaving a multi-tap structure, a row scanning circuit configured togenerate a plurality of sampling control signals applied to a pluralityof taps of the depth pixel, and a controller configured to control thelight source, the pixel array and the row scanning circuit based on amode signal indicating a selected operation mode of a ToF sensor among adistance detection mode to sense a distance to an object and a pluralityof additional operation modes such that the plurality of taps generate aplurality of sample data corresponding to the selected operation mode.

According to an aspect of another example embodiment, a method ofoperating a time-of-flight (ToF) sensor including at least one depthpixel having a multi-tap structure and a light source illuminating atransmission light to an object, includes, determining an operation modeof a ToF sensor, from among a distance detection mode to sense adistance to an object and a plurality of additional operation modes,based on the determined operation mode being the distance detectionmode, applying a plurality of sampling control signals of differentphases to a plurality of taps of a depth pixel during an integrationperiod to collect a photo charge generated by an incident light, andbased on the determined operation mode being one of the plurality ofadditional operation modes, dividing the integration period into aplurality of shot periods and selectively activating the transmissionlight and the plurality of sampling control signals during the pluralityof shot periods based on the selected operation mode.

According to an aspect of another example embodiment, an apparatus foroperating a time-of-flight sensor includes a memory storinginstructions, and at least one processor configured to execute theinstructions to determine an operation mode of a ToF sensor, from amonga distance detection mode to sense a distance to an object and aplurality of additional operation modes, based on the determinedoperation mode being the distance detection mode, control to apply aplurality of sampling control signals of different phases to a pluralityof taps of a depth pixel during an integration period to collect a photocharge generated by an incident light, and based on the determinedoperation mode being one of the plurality of additional operation modes,divide the integration period into a plurality of shot periods andcontrol to selectively activate the transmission light and the pluralityof sampling control signals during the plurality of shot periods basedon the determined operation mode.

According to an aspect of another example embodiment, a non-transitorycomputer-readable recording medium has recorded thereon instructionsexecutable by at least one processor to perform a method of operating aToF sensor, the method including determining an operation mode of a ToFsensor, from among a distance detection mode to sense a distance to anobject and a plurality of additional operation modes, based on thedetermined operation mode being the distance detection mode, applying aplurality of sampling control signals of different phases to a pluralityof taps of a depth pixel during an integration period to collect a photocharge generated by an incident light, and based on the determinedoperation mode being one of the plurality of additional operation modes,dividing the integration period into a plurality of shot periods andselectively activating the transmission light and the plurality ofsampling control signals during the plurality of shot periods based onthe determined operation mode.

The ToF sensor and the method of operating the ToF sensor according toexample embodiments may perform a plurality of functions, in addition toan original function of the ToF sensor to measure a distance, bycontrolling the plurality of taps of the depth pixel and the lightsource depending on the operation modes.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments of the present disclosure will be more clearlyunderstood from the following detailed description taken in conjunctionwith the accompanying drawings, in which:

FIG. 1 is a flowchart illustrating a method of operating atime-of-flight (ToF) sensor according to an example embodiment;

FIG. 2 is a block diagram illustrating a ToF sensor according to anexample embodiment;

FIGS. 3 and 4 are diagrams for describing an example method of measuringand calculating a distance to an object;

FIG. 5 is a circuit diagram illustrating an example embodiment of adepth pixel including a ToF sensor according to an example embodiment;

FIG. 6 is a timing diagram illustrating an operation of a ToF sensor ina distance detection mode according to an example embodiment;

FIG. 7 is a flowchart illustrating a method of operating a ToF sensor inan object detection mode according to an example embodiment;

FIG. 8 is a timing diagram illustrating an operation of a ToF sensor inan objection detection mode according to an example embodiments;

FIG. 9 is a flowchart illustrating a method of operating a ToF sensor ina motion detection mode according to an example embodiment;

FIGS. 10 through 13 are timing diagrams illustrating an operation of aToF sensor in a motion detection mode according to example embodiments;

FIG. 14 is a flowchart illustrating a method of operating a ToF sensorin a combination detection mode according to an example embodiment;

FIGS. 15 and 16 are timing diagrams illustrating an operation of a ToFsensor in a combination detection mode according to example embodiments;

FIG. 17 is a flowchart illustrating a method of operating a ToF sensorin a wide dynamic range (WDR) mode according to an example embodiment;

FIGS. 18 through 21 are timing diagrams illustrating an operation of aToF sensor in a WDR mode according to example embodiments;

FIG. 22 is a circuit diagram illustrating a depth pixel having afour-tap structure according to an example embodiment;

FIG. 23 is a timing diagram illustrating an example embodiment of anoperation of a depth pixel in a distance detection mode;

FIGS. 24 and 25 are diagrams illustrating example embodiments of astructure of sharing a floating diffusion region of a depth pixel;

FIG. 26 is a block diagram illustrating an electronic device accordingto an example embodiment; and

FIG. 27 is a block diagram illustrating a camera module included in theelectronic device of FIG. 26.

DETAILED DESCRIPTION

Various example embodiments will be described more fully hereinafterwith reference to the accompanying drawings, in which some exampleembodiments are shown. In the drawings, like numerals refer to likeelements throughout. The repeated descriptions may be omitted.Hereinafter, it is understood that expressions such as “at least oneof,” when preceding a list of elements, modify the entire list ofelements and do not modify the individual elements of the list. Forexample, the expressions “at least one of [A], [B], and [C]” or “atleast one of [A], [B], or [C]” means only A, only B, only C, A and B, Band C, A and C, or A, B, and C.

FIG. 1 is a flowchart illustrating a method of operating atime-of-flight (ToF) sensor according to an example embodiment.

Referring to FIG. 1, to operate a time-of-flight (ToF) sensor includingat least one depth pixel having a multi-tap structure and a light sourceilluminating a transmission light to an object, a selected operationmode of a ToF sensor is determined among a distance detection mode tosense a distance to an object and a plurality of additional operationmodes (S100) (or at least one additional operation mode). The depthpixel, the ToF sensor and the distance detection mode, which maycorrespond to the original function of the ToF sensor, are describedbelow with reference to FIGS. 2 through 6.

According to example embodiments, the plurality of additional operationmodes may include at least one of an object detection mode to senseexistence of the object, a motion detection mode to sense a motion ofthe object, a combination detection mode to simultaneously sense theexistence of the object and the motion of the object, and a wide dynamicrange (WDR) mode to sense the object with a plurality of sensingsensitivities. The plurality of additional operation modes according toexample embodiments are described below with reference to FIGS. 7through 21.

A plurality of taps of a depth pixel and a light source are controlledbased on the selected operation mode such that the plurality of tapsgenerate a plurality of sample data corresponding to the selectedoperation mode (S200).

In the distance detection mode, during an integration period to collecta photo charge generated by an incident light, the transmission lightmay be generated using the light source and the distance may be measuredusing a plurality of sampling control signals of different phases. Incontrast, in at least one (or all) of the plurality of additionaloperation modes, the integration period may be divided into a pluralityof shot periods, and the plurality of sampling control signals may beselectively activated during the plurality of shot periods based on theselected operation mode.

A sensing result corresponding to the selected operation mode isdetermined based on the plurality of sample data (S300). The pluralityof sample data may include different information depending on theoperation modes. A method of determining the sensing result according toexample embodiments is described below with reference to FIGS. 7 through21.

As such, the ToF sensor and the method of operating the ToF sensoraccording to an example embodiment may perform a plurality of functions,in addition to a function of the ToF sensor to measure a distance, bycontrolling the plurality of taps of the depth pixel and the lightsource depending on the operation modes.

Hereinafter, configuration and operation of the ToF sensor are describedwith reference to FIGS. 2 through 6, and additional functions using theToF sensor are described below with reference to FIGS. 7 through 21.

FIG. 2 is a block diagram illustrating a ToF sensor 100 according to anexample embodiment.

Referring to FIG. 2, a ToF sensor 100 includes a sensing unit (e.g.,sensor), a controller 150 and a light source module 200 (e.g., lightsource). The sensing unit may include a pixel array 110, ananalog-to-digital converter (ADC) unit 120 (or ADC), a row scanningcircuit 130, and a column scanning circuit 140.

The pixel array 110 may include depth pixels receiving light RL that isreflected from an object OBJ after being transmitted to the object OBJby the light source module 200. The depth pixels may convert thereception light RL into electrical signals. The depth pixels may provideinformation about a distance of the object OBJ from the ToF sensor 100and/or black-and-white image information.

The pixel array 110 may further include color pixels for providing colorimage information. In this case, the ToF sensor 100 may be athree-dimensional color image sensor that provides the color imageinformation and the depth information. According to example embodiments,an infrared filter and/or a near-infrared filter may be formed on thedepth pixels, and a color filter (e.g., red, green and blue filters) maybe formed on the color pixels. According to example embodiments, a ratioof the number of the depth pixels to the number of the color pixels mayvary as desired or by design.

The ADC unit 120 may convert an analog signal output from the pixelarray 110 into a digital signal. According to example embodiments, theADC unit 120 may perform a column analog-to-digital conversion thatconverts analog signals in parallel using a plurality ofanalog-to-digital converters respectively coupled to a plurality ofcolumn lines. According to example embodiments, the ADC unit 120 mayperform a single analog-to-digital conversion that sequentially convertsthe analog signals using a single analog-to-digital converter.

According to example embodiments, the ADC unit 120 may further include acorrelated double sampling (CDS) unit for extracting an effective signalcomponent. The CDS unit may perform an analog double sampling thatextracts the effective signal component based on a difference between ananalog reset signal including a reset component and an analog datasignal including a signal component. Further, the CDS unit may perform adigital double sampling that converts the analog reset signal and theanalog data signal into two digital signals and extracts the effectivesignal component based on a difference between the two digital signals.Additionally, the CDS unit may perform a dual correlated double samplingthat performs both the analog double sampling and the digital doublesampling.

The row scanning circuit 130 may receive control signals from thecontroller 150, and may control a row address and a row scan of thepixel array 110. To select a row line among a plurality of row lines,the row scanning circuit 130 may apply a signal for activating theselected row line to the pixel array 110. According to exampleembodiments, the row scanning circuit 130 may include a row decoder thatselects a row line of the pixel array 110 and a row driver that appliesa signal for activating the selected row line.

The column scanning circuit 140 may receive control signals from thecontroller 150, and may control a column address and a column scan ofthe pixel array 110. The column scanning circuit 140 may output adigital output signal from the ADC unit 120 to a digital signalprocessing circuit (not shown) and/or to an external host (not shown).For example, the column scanning circuit 140 may provide the ADC unit120 with a horizontal scan control signal to sequentially select aplurality of analog-to-digital converters included in the ADC unit 120.

The controller 150 may control the ADC unit 120, the row scanningcircuit 130, the column scanning circuit 140, and the light sourcemodule 200. The controller 150 may provide the ADC unit 120, the rowscanning circuit 130, the column scanning circuit 140, and the lightsource module 200 with control signals, such as at least one of a clocksignal, a timing control signal, or the like. The controller 150 mayinclude at least one of a control logic circuit, a phase locked loopcircuit, a timing control circuit, a communication interface circuit, orthe like.

The light source module 200 may emit light of a desired (or,alternatively, a predetermined) wavelength. For example, the lightsource module 200 may emit infrared light and/or near-infrared light.The light source module 200 may include a light source 210 and a lens220. The light source 210 may be controlled by the controller 150 toemit the transmission light TL of a desired intensity and/orcharacteristic (for example, periodic). For example, the intensityand/or characteristic of the transmission light TL may be controlledsuch that the transmission light TL has a waveform of a pulse wave, asine wave, a cosine wave, or the like. The light source 210 may beimplemented by a light emitting diode (LED), a laser diode, or the like.

Hereinafter, a first operation (e.g., normal operation) of the ToFsensor 100 according to example embodiments is described below.

The controller 150 may control the light source module 200 to emit thetransmission light TL having the periodic intensity. The transmissionlight TL emitted by the light source module 200 may be reflected fromthe object OBJ back to the ToF sensor 100 as the reception light RL. Thereception light RL may be incident on the depth pixels, and the depthpixels may be activated by the row scanning circuit 130 to output analogsignals corresponding to the reception light RL. The ADC unit 120 mayconvert the analog signals output from the depth pixels into sample dataSDATA. The sample data SDATA may be provided to the controller 150 bythe column scanning circuit 140 and/or the ADC 120.

The controller 150 may calculate a distance of the object OBJ from theToF sensor 100, a horizontal position of the object OBJ, a verticalposition of the object OBJ and/or a size of the object OBJ based on thesample data SDATA. The controller 150 may control the emission angle ora projection (or incident) region of the transmission light TL based onthe distance, the horizontal position, the vertical position and/or thesize of the object OBJ. For example, the controller 150 may control aninterval between the light source 210 and the lens 220, a relativeposition (or a placement) of the light source 210 and the lens 220 withrespect to each other, a refractive index of the lens 220, a curvatureof the lens 220, or the like.

The transmission light TL illuminated to the object OBJ may be reflectedand the reflection light RL may be incident on the depth pixels in thepixel array 110. The depth pixels may output analog signalscorresponding to the reflection light RL, the ADC unit 120 may convertthe analog signals to digital data or the sample data SDATA. The sampledata SDATA and/or the depth information may be provided to thecontroller 150, the digital signal processing circuit and/or theexternal host. According to example embodiments, the pixel array 110 mayinclude color pixels, and color image information as well as the depthinformation may be provided to the digital signal processing circuitand/or the external host.

The external host or processor may determine the selected operation modeof the ToF sensor 100 according to various scenarios, and provide a modesignal MD indicating the selected operation mode to the ToF sensor 100.Based on the mode signal MD, the controller 150 of the ToF sensor 100may control the light source 210, the pixel array 110 and the rowscanning circuit 130 to perform an operation corresponding to theselected operation mode.

In some example embodiments, the controller 150 or the external host maychange the selected operation mode based on the sensing result of thepresent selected operation mode. For example, the ToF sensor 100 mayoperate in one of the additional operation modes and, if necessary,desired, or controlled, the selected operation mode may be converted tothe distance detection mode. In some example embodiments, the selectedoperation mode may be changed step-by-step, for example, from the objectdetection mode to the motion detection mode, and from the motiondetection mode to the distance detection mode.

FIGS. 3 and 4 are diagrams for describing an example method of measuringand calculating a distance to an object.

Referring to FIGS. 2 and 3, the transmission light TL emitted by a lightsource module 200 may have a periodic intensity and/or characteristic.For example, the intensity (i.e., the number of photons per unit area)of the transmission light TL may have a waveform of a sine wave.

The transmission light TL emitted by the light source module 200 may bereflected from the object OBJ, and then may be incident on the pixelarray 110 as reception light RL. The pixel array 110 may periodicallysample the reception light RL. According to example embodiments, duringeach period of the reception light RL (for example, corresponding to aperiod of the transmitted light TL), the pixel array 110 may perform asampling on the reception light RL by sampling, for example, at twosampling points having a phase difference of about 180 degrees, at foursampling points having a phase difference of about 90 degrees, or atmore than four sampling points. For example, the pixel array 110 mayextract four samples A0, A1, A2 and A3 of the reception light RL atphases of 90 degrees (or about 90 degrees), 180 degrees (or about 180degrees), 270 degrees (or about 270 degrees) and 360 degrees (or about360 degrees) per period, respectively.

The reception light RL may have an offset B that is different from anoffset of the transmission light TL emitted by the light source module200 due to background light, a noise, or the like. The offset B of thereception light RL may be calculated by Equation 1.

$\begin{matrix}{B = \frac{{A\; 0} + {A\; 1} + {A\; 2} + {A\; 3}}{4}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

Here, A0 represents an intensity of the reception light RL sampled at aphase of about 90 degrees of the emitted light TL, A1 represents anintensity of the reception light RL sampled at a phase of about 180degrees of the emitted light TL, A2 represents an intensity of thereception light RL sampled at a phase of about 270 degrees of theemitted light TL, and A3 represents an intensity of the reception lightRL sampled at a phase of about 360 degrees of the emitted light TL.

The reception light RL may have an amplitude A lower than that of thetransmission light TL emitted by the light source module 200 due to loss(for example, light loss). The amplitude A of the reception light RL maybe calculated by Equation 2.

$\begin{matrix}{A = \frac{\sqrt{\left( {{A\; 0} - {A\; 2}} \right)^{2} + \left( {{A\; 1} - {A\; 3}} \right)^{2}}}{2}} & \left\lbrack {{Equation}\mspace{14mu} 2} \right\rbrack\end{matrix}$

Black-and-white image information about the object OBJ may be providedby respective depth pixels included in the pixel array 110 based on theamplitude A of the reception light RL.

The reception light RL may be delayed by a phase difference Φcorresponding, for example, to twice the distance of the object OBJ fromthe t ToF sensor 100 with respect to the emitted light TL. The phasedifference Φ between the emitted light TL and the reception light RL maybe calculated by Equation 3.

$\begin{matrix}{\phi = {\arctan\left( \frac{{A\; 0} - {A\; 2}}{{A\; 1} - {A\; 3}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack\end{matrix}$

The phase difference Φ between the emitted light TL and the receptionlight RL may, for example, correspond to a time-of-flight (TF). Thedistance of the object OBJ from the ToF sensor 100 may be calculated byan equation, “R=c*TF/2”, where R represents the distance of the objectOBJ, and c represents the speed of light. Further, the distance of theobject OBJ from the ToF sensor 100 may also be calculated by Equation 4using the phase difference Φ between the emitted light TL and thereception light RL.

$\begin{matrix}{R = {\frac{c}{4\pi\; f}\phi}} & \left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack\end{matrix}$

Here, f represents a modulation frequency, which is a frequency of theintensity of the emitted light TL (or a frequency of the intensity ofthe reception light RL).

As described above, the ToF sensor 100 according to example embodimentsmay obtain depth information about the object OBJ using the transmissionlight TL emitted by the light source module 200. Although FIG. 3illustrates the transmission light TL of which the intensity has awaveform of a sine wave, it is understood that one or more otherembodiments are not limited thereto. For example, the ToF sensor 100 mayuse the transmission light TL of which the intensity has various typesof waveforms, according to example embodiments. Further, the ToF sensor100 may extract the depth information according to the waveform of theintensity of the transmission light TL, a structure of a depth pixel, orthe like.

FIG. 4 illustrates an example of modulation timing and demodulationtimings of a depth pixel having a four-tap structure, whereby theoperation of the ToF sensor may vary.

Referring to FIG. 4, the transmission light TL from the light source 210may be output in synchronization with a signal provided from thecontroller 150. The first through fourth demodulation signals DEM1through DEM4 may be generated in synchronization with the signal fromthe controller 150. The first through fourth demodulation signals DEM1through DEM4 have phase differences of 0, 90, 180 and 270 degrees,respectively. As described above with reference to FIG. 3, four samplesA0, A1, A2 and A3 of the reception light RL may be sampled at phases ofabout 90 degrees, about 180 degrees, about 270 degrees and about 360degrees per period, respectively.

FIG. 4 illustrates an example that the phase of the first demodulationsignal DEM1 coincides with the phase of the transmission light TL. Thefirst through fourth demodulation signals DEM1 through DEM4 may beapplied to first through fourth demodulation transfer gates,respectively, as described below.

While FIGS. 3 and 4 are provided for describing the principal ofmeasuring and calculating a distance to an object using a ToF sensor, itis understood that one or more other example embodiments are not limitedthereto. The duty ratio of the transmission light TL and the number, thephase differences and the duty ratio of the demodulation signals mayvary.

Hereinafter, a depth pixel having a four-tap structure and operation ofthe depth pixel are described with reference to FIGS. 5 and 6. It isunderstood that one or more other example embodiments are not limited tothe four-tap structure, and may be applied to arbitrary depth pixelshaving three or more taps.

FIG. 5 is a circuit diagram illustrating an example embodiment of adepth pixel including a ToF sensor.

Referring to FIG. 5, the depth pixel PX1 may include a first photogatePGA and transistors TMA, TS1 and TT1 corresponding to a first tap TA, asecond photogate PGB and transistors TMB, TS1 and TT1 corresponding to asecond tap TB, a third photogate PGC and transistors TMC, TS2 and TT2corresponding to a third tap TC, a fourth photogate PGD and transistorsTMD, TS2 and TT2 corresponding to a fourth tap TD, transistors TRS1,TRS2, TSF1, TSF2, TSL1 and TSL2 corresponding to a readout circuit, andoverflow gates OG and a photodiode corresponding to a shared circuit.

Each of the transistors TMA, TMB, TMC, TMD, TS1, TS2, TT1, TT2, TRS1 andTRS2 may include a gate disposed above a semiconductor substrate andsource and drain regions disposed at both sides of the gate in thesemiconductor substrate. The gates of the transistors TMA, TMB, TMC,TMD, TS1, TS2, TT1, TT2, TRS1 correspond to a first demodulationtransfer gate TGA, a second demodulation transfer gate TGB, a thirddemodulation transfer gate TGC, a fourth demodulation transfer gate TGD,storage gates SG, FD transfer gates TG and reset gates RG1 and RG2,respectively.

First through fourth photogate signals SPGA˜SPGD are applied to thefirst through fourth photogates PGA˜PGD, an overflow gate voltage VOG isapplied to the overflow gates OG storage control signals SSG1 and SSG2are applied to the storage gates SG1 and SG2, demodulation transfercontrol signals STGA˜STGD are applied to the demodulation transfer gatesTGA˜TGD, FD transfer control signals STG1 and STG2 are applied to the FDtransfer gates TG1 and TG2, reset signals SRG1 and SRG2 are applied tothe reset gates RG1 and RG2, and selection signals SEL1 and SEL2 areapplied to the gates of the selection transistors TSL1 and TSL2. Thephotogate signals SPGA˜SPGD, the overflow gate voltage VOG, thedemodulation transfer control signals STGA˜STGD, the storage controlsignals SSG1 and SSG2, the FD transfer control signals STG1 and STG2,the reset signals SRG1 and SRG2, and the selection signals SEL1 and SEL2may be provided from the row scanning circuit 130 under control of thecontroller 150 as described above with reference to FIG. 2.

The first through fourth photogate signals SPGA˜SPGD correspond to thesampling control signals as described above with reference to FIG. 1.The row scanning circuit 130 may generate the first through fourthphotogate signals SPGA˜SPGD corresponding to the sampling controlsignals based on the selected operation mode as described below.

The storage gates SG1 and SG2 are one of charge storing structures totemporarily store the photo charge transferred from the common photogateCPG before transferring the photo charge to the floating diffusionregions FDA, FDB, FDC and FDD. In some example embodiments, the chargestoring structure may be implemented with the storage gates SG1 and SG2alone. In other example embodiments, the charge storing structure may beimplemented with the storage gates SG1 and SG2 and storage diodes formedin the semiconductor substrate under the storage gates SG1 and SG2.Using such a charge storing structure, the true correlated doublesampling (CDS) may be performed and noise in the readout signals may bereduced. According to example embodiments, the FD transfer gates TG1 andTG2 and/or the storage gates SG1 and SG2 may be omitted.

The charge stored in the floating diffusion regions FDA, FDB, FDC andFDD may be provided as output signals, that is, the sample data SOA˜SOD,using the source follower transistors TSF1 and TSF2 and the selectiontransistors TSL1 and TSL2.

FIG. 6 is a timing diagram illustrating an operation of a ToF sensor ina distance detection mode according to an example embodiment.

Referring to FIGS. 2, 5 and 6, in the distance detection mode, the lightsource 210 may generate the transmission light TL modulated by amodulation frequency during an integration period TINT to collect aphoto charge generated by an incident light. The row scanning circuit130 may apply the first through fourth sampling control signals, thatis, the first through fourth photogate signals SPGA˜SPGD having thedifferent phases, to the first through fourth photogates PGA˜PGDcorresponding to the first through fourth taps TA˜TD. In some exampleembodiments, the phase of the first photogate signal STGA may besynchronized with the phase of the transmission light TL. In someexample embodiments, the phase difference between the first and secondphotogate signals STGA and STGB may be 90 degrees, the phase differencebetween the first and third photogate signals STGA and STGC may be 180degrees, and the phase difference between the first and fourth photogatesignals STGA and STGD may be 270 degrees. Using the photogate signalsSTGA˜STGB having the different phases, the distance to the object OBJmay be measured as described above with reference to FIGS. 3 and 4.

The overflow voltage VOG applied to the overflow gates OG may have aturn-off voltage level to block the photo charge from being drained fromthe photodiode PD during the integration period TINT. The demodulationtransfer control signals STGA˜STGD and the storage control signals SSG1and SSG2 are activated during the integration period TINT. Accordingly,the photo charge collected by the first through fourth photogate signalsSPGA˜SPGD may be stored in the semiconductor substrate under the storagegates SG1 and SG2, respectively.

During the other periods, for example, a reset period TRST to initializethe depth pixel PX1 and a readout period TRD to measure an amount of thephoto charge collected during the integration period TINT, the overflowgate voltage VOG may have a turn-on voltage level VON to drain the photocharge from the photodiode PD. The collected photo charge may be drainedto the terminal of the power supply voltage VDD during the periods TRSTand TRD other than the integration period TINT. As such, a globalshutter function may be implemented using the overflow gates OG.

At a first time point t10 during the readout period TRD when the resetsignal SRG1 is deactivated and the selection signal SEL1 is activated,first and second reset state data of the first and second taps TA and TBmay be output through the column lines COL1 and COL2, respectively. At asecond time point t11 during the readout period TRD when the FD transfercontrol signal STG1 is activated and the storage control signal SSG1 isdeactivated, the photo charge stored by the storage gates SG may betransferred to the floating diffusion regions FDA and FDB and the firstand second sample data SOA and SOB of the first and second taps TA andTB may be output through the column lines COL1 and COL2, respectively.

At a third time point t20 during the readout period TRD when the resetsignal SRG2 is deactivated and the selection signal SEL2 is activated,third and fourth reset state data of the third and fourth taps TC and TDmay be output through the column lines COL1 and COL2, respectively. At afourth time point t21 during the readout period TRD when the FD transfercontrol signal STG2 is activated and the storage control signal SSG2 isdeactivated, the photo charge stored by the storage gates SG may betransferred to the floating diffusion regions FDC and FDD and the thirdand fourth sample data SOC and SOD of the third and fourth taps TC andTD may be output through the column lines COL1 and COL2, respectively.

Hereinafter, redundant descriptions may be omitted with respect totiming diagrams of FIGS. 8, 10, 11, 12, 13, 15, 16, 18, 19, 20 and 21 inwhich the signals other than the photogate signals SPGA˜SPGD are thesame as described with reference to FIG. 6.

FIG. 7 is a flowchart illustrating a method of operating a ToF sensor inan object detection mode according to an example embodiment, and FIG. 8is a timing diagram illustrating an operation of a ToF sensor in anobjection detection mode according to an example embodiment.

Referring to FIGS. 7 and 8, when the selected operation mode is anobject detection mode to sense the existence of an object, theintegration period TINT may be divided into a first shot period TS1 anda second shot period TS2 (S211)

The transmission light TL may be deactivated during the first shotperiod TS1 (S212), and ambient light sample data corresponding to anambient light may be generated by activating at least one first signalamong a plurality of sampling control signals to control the pluralityof taps during the first shot period TS1 (S213). For example, asillustrated in FIG. 8, the photogate signals SPGA and SPGD correspondingto the at least one first signal applied to the first and fourth taps TAand TD may be activated during the first shot period TS1, and the firstsample data SOA and the fourth sample data SOD may correspond to theambient light sample data.

The transmission light TL may be activated during the second shot periodTS2 (S214), and object sample data corresponding to the object may begenerated by activating at least one second signal among the pluralityof sampling control signals during the second shot period TS2 (S215).For example, as illustrated in FIG. 8, the photogate signals SPGB andSPGC corresponding to the at least one second signal applied to thesecond and third taps TB and TC may be activated during the second shotperiod TS2, and the second sample data SOB and the third sample data SOCmay correspond to the object sample data.

As such, the integration period TINT may be divided into the first shotperiod TS1 and the second shot period TS2 and the multi-tap operationmay be performed using the transmission light TL including a singlepulse during the second shot period TS2. Through the operation duringthe readout period TRD as described with reference to FIG. 6, theambient light sample data SOA and SOD and the object sample data SOB andSOC may be provided.

The sensing result corresponding to the object detection mode may bedetermined based on the ambient light sample data SOA and SOD and theobject sample data SOB and SOC. In some example embodiments, whether theobject exists within a reference distance may be determined based on avalue SOB+SOC−SOA−SOD of the object sample data SOB and SOC subtractedby the ambient light sample data SOA and SOD. For example, it may bedetermined that the object exists within the reference distance if thevalue SOB+SOC−SOA−SOD is greater than a reference value corresponding tothe reference distance. In some example embodiments, the readout of thereset state data at the first and third time points t10 and t20 asdescribed above with reference to FIG. 6 may be omitted to increase thereadout speed. In this case the CDS is not performed, but the resetdeviations of the taps TA˜TD may be canceled to some extent in the valueSOB+SOC−SOA−SOD itself and the advantage of the high-speed readout maybe adopted.

FIG. 9 is a flowchart illustrating a method of operating a ToF sensor ina motion detection mode according to an example embodiment, and FIGS. 10through 13 are timing diagrams illustrating an operation of a ToF sensorin a motion detection mode according to one or more example embodiments.

Referring to FIGS. 9, 10 and 11, when the selected operation mode is amotion detection mode to sense a motion of the object, the integrationperiod TINT may be divided into a first shot period TS1 and a secondshot period TS2 (S221).

First object sample data corresponding to the object may be generated byactivating at least one first signal among a plurality of samplingcontrol signals to control the plurality of taps during the first shotperiod TS1 (S222). For example, as illustrated in FIGS. 10 and 11, thephotogate signals SPGA and SPGD corresponding to the at least one firstsignal applied to the first and fourth taps TA and TD may be activatedduring the first shot period TS1, and the first sample data SOA and thefourth sample data SOD may correspond to the first object sample data.

Second object sample data corresponding to the object may be generatedby activating at least one second signal among the plurality of samplingcontrol signals during the second shot period TS2 (S223). For example,as illustrated in FIGS. 10 and 11, the photogate signals SPGB and SPGCcorresponding to the at least one second signal applied to the secondand third taps TB and TC may be activated during the second shot periodTS2, and the second sample data SOB and the third sample data SOC maycorrespond to the second object sample data.

In some example embodiments, as illustrated in FIG. 10, the light source210 in FIG. 2 may be controlled to activate the transmission light TL toinclude a single pulse during the integration period TINT. In this case,the first object sample data SOA and SOD and the second object sampledata SOB and SOC correspond to active signals related to thetransmission light TL.

In some example embodiments, as illustrated in FIG. 11, the light source210 in FIG. 2 may be controlled to deactivate the transmission light TLduring the integration period TINT. In this case, the first objectsample data SOA and SOD and the second object sample data SOB and SOCcorrespond to passive signals irrelevant to the transmission light TL.

The sensing result corresponding to the motion detection mode may bedetermined based on the first object sample data SOA and SOD and thesecond object sample data SOB and SOC. In some example embodiments, themotion of the object may be determined based on a difference valueSOB+SOC-SOA-SOD between the second object sample data SOB and SOC andthe first object sample data SOA and SOD. For example, it may bedetermined that the object moves toward the ToF sensor if the differencevalue SOB+SOC-SOA-SOD is a positive value. In contrast, it may bedetermined that the object moves far away from the ToF sensor if thedifference value SOB+SOC-SOA-SOD is a negative value. In some exampleembodiments, the readout of the reset state data at the first and thirdtime points t10 and t20 as described above with reference to FIG. 6 maybe omitted to increase the readout speed. In this case, the CDS is notperformed, but the reset deviations of the taps TA˜TD may be canceled tosome extent in the value SOB+SOC-SOA-SOD itself and the advantage of thehigh-speed readout may be adopted.

According to example embodiments, the integration period TINT may bedivided into a plurality of shot periods, e.g., first through fourthshot periods TS1˜TS4 as illustrated in FIGS. 12 and 13. A plurality ofobject sample data SOA˜SOD corresponding to the object may be generatedby sequentially activating a plurality of sampling control signals, thatis, the first through fourth photogate signals SPGA˜SPGD, to control theplurality of taps TA˜TD during the plurality of shot periods TS1˜TS4.When detecting high-speed motion, the integration period TINT may bedivided into more shot periods to obtain further subdivided objectsample data.

FIG. 14 is a flowchart illustrating a method of operating a ToF sensorin a combination detection mode according to an example embodiment, andFIGS. 15 and 16 are timing diagrams illustrating an operation of a ToFsensor in a combination detection mode according to one or more exampleembodiments.

Referring to FIGS. 14 and 15, when (or based on) the selected operationmode is a combination detection mode to simultaneously sense anexistence of the object and a motion of the object, the integrationperiod TINT may be divided into a first shot period TS1, a second shotperiod TS2 and a third shot period TS3 (S231).

The transmission light TL may be deactivated during the first shotperiod TS1 (S232), and ambient light sample data corresponding to anambient light may be generated by activating at least one first signalamong a plurality of sampling control signals to control the pluralityof taps during the first shot period TS1 (S233). For example, asillustrated in FIG. 15, the photogate signals SPGA and SPGDcorresponding to the at least one first signal applied to the first andfourth taps TA and TD may be activated during the first shot period TS1,and the first sample data SOA and the fourth sample data SOD maycorrespond to the ambient light sample data or ambient signals.

The transmission light TL may be activated during the second shot periodTS2 and the third shot period TS3 (S234).

First object sample data corresponding to the object may be generated byactivating at least one second signal among the plurality of samplingcontrol signals during the second shot period TS2 (S235). For example,as illustrated in FIG. 15, the photogate signal SPGB corresponding tothe at least one second signal applied to the second tap TB may beactivated during the second shot period TS2, and the second sample dataSOB may correspond to the first object sample data corresponding to theactive signal related to the transmission light TL.

Second object sample data corresponding to the object may be generatedby activating at least one third signal among the plurality of samplingcontrol signals during the third shot period TS3 (S236). For example, asillustrated in FIG. 15, the photogate signal SPGC corresponding to theat least one third signal applied to the third tap TC may be activatedduring the third shot period TS3, and the third sample data SOC maycorrespond to the second object sample data corresponding to the activesignal related to the transmission light TL.

According to example embodiments, at least one of the plurality ofsampling control signals may be deactivated during the integrationperiod TINT to generate noise sample data indicating the sensing noiseof the depth pixel. For example, as illustrated in FIG. 16, thephotogate signal SPGD applied to the fourth tap TD may be deactivatedduring the integration period TINT and the fourth sample data SOD may beused as the noise sample data or a noise signal. The sensing noise ofthe depth pixel may include a dark noise, a light leakage noise, anoffset noise, etc.

The sensing result corresponding to the object detection mode may bedetermined based on the value SOB+SOC-SOA-SOD of the first and secondobject sample data SOB and SOC subtracted by the ambient light sampledata SOA and SOD, as described above with reference to FIGS. 7 and 8. Inaddition, the sensing result corresponding to the motion detection modemay be determined based on the difference value SOC−SOB between thefirst object sample data SOB and the second object sample data SOC, asdescribed above with reference to FIGS. 9 through 13. In the exampleembodiment of FIG. 16, the further finite sensing result may be obtainedby compensating for the ambient light sample data SOA, the first objectsample data SOB and the second object sample data SOC using the noisesample data SOD.

FIG. 17 is a flowchart illustrating a method of operating a ToF sensorin a wide dynamic range (WDR) mode according to an example embodiment,and FIGS. 18 through 21 are timing diagrams illustrating an operation ofa ToF sensor in a WDR mode according to one or more example embodiments.

The WDR scheme is used to simultaneously capture a dark portion of animage and a bright portion of the image. The problems of backlight maybe solved and the clearer image may be obtained by combining the data ofthe higher sensing sensitivity in the dark portion and the data of thelower sensing sensitivity in the bright portion. The WDR scheme mayprovide a better result than the backlight compensation (BLC) scheme.The dynamic range indicates a ratio between the brightest portion andthe darkest portion within the distinguishable range.

Referring to FIGS. 17 through 21, when the selected operation mode is aWDR mode to sense the object with a plurality of sensing sensitivities,the integration period TINT may be divided into a first shot period TS1,a second shot period TS2 longer than the first shot period TS1 and athird shot period TS3 longer than the second shot period TS2 (S241).

First sensitivity sample data may be generated by activating at leastone first signal among a plurality of sampling control signals tocontrol the plurality of taps during the first shot period TS1 (S242).For example, as illustrated in FIGS. 18 through 21, the photogate signalSPGA corresponding to the at least one first signal applied to the firsttap TA may be activated during the first shot period TS1, and the firstsample data SOA may correspond to the first sensitivity sample data of ashort exposure.

Second sensitivity sample data may be generated by activating at leastone second signal among the plurality of sampling control signals duringthe second shot period TS2 (T243). For example, as illustrated in FIGS.18 through 21, the photogate signal SPGB corresponding to the at leastone second signal applied to the second tap TB may be activated duringthe second shot period TS2, and the second sample data SOB maycorrespond to the second sensitivity sample data of a middle exposure.

Third sensitivity sample data may be generated by activating at leastone third signal among the plurality of sampling control signals duringthe third shot period TS3 (S244). For example, as illustrated in FIGS.18 and 19, the photogate signals SPGC and SPGD corresponding to the atleast one third signal applied to the third and fourth taps TC and TDmay be activated during the third shot period TS3, and the third andfourth sample data SOC and SOD may correspond to the third sensitivitysample data of a long exposure.

According to example embodiments, at least one of the plurality ofsampling control signals may be deactivated during the integrationperiod TINT to generate noise sample data indicating the sensing noiseof the depth pixel. For example, as illustrated in FIGS. 20 and 21, thephotogate signal SPGD applied to the fourth tap TD may be deactivatedduring the integration period TINT and the fourth sample data SOD may beused as the noise sample data or a noise signal. The sensing noise ofthe depth pixel may include a dark noise, a light leakage noise, anoffset noise, etc.

In some example embodiments, as illustrated in FIGS. 18 and 20, thelight source 210 in FIG. 2 may be controlled to activate thetransmission light TL to include a single pulse during the integrationperiod TINT. In some example embodiments, as illustrated in FIGS. 19 and21, the light source 210 in FIG. 2 may be controlled to deactivate thetransmission light TL during the integration period TINT.

FIG. 22 is a circuit diagram illustrating a depth pixel PX2 having afour-tap structure according to an example embodiment, and FIG. 23 is atiming diagram illustrating an example embodiment of an operation of adepth pixel in a distance detection mode. Hereinafter, descriptionsredundant with those provided above with reference to FIGS. 5 and 6 maybe omitted.

Referring to FIG. 22, a depth pixel PX2 may include transistors TMA, TS1and TT1 corresponding to a first tap TA, transistors TMB, TS1 and TT1corresponding to a second tap TB, transistors TMC, TS2 and TT2corresponding to a third tap TC, transistors TMD, TS2 and TT2corresponding to a fourth tap TD, transistors TRS1, TRS1, TSF1, TSF2,TSL1 and TSL2 corresponding to a readout circuit, and a common photogateCPC; overflow gates OG1 and OG2 and a photodiode PD corresponding to ashared circuit.

Each of the transistors TMA, TMB, TMC, TMD, TS1, TS2, TT1, TT2, TRS1 andTRS2 may include a gate disposed above a semiconductor substrate andsource and drain regions disposed at both sides of the gate in thesemiconductor substrate. The gates of the transistors TMA, TMB, TMC,TMD, TS1, TS2, TT1, TT2, TRS1 correspond to a first demodulationtransfer gate TGA, a second demodulation transfer gate TGB, a thirddemodulation transfer gate TGC, a fourth demodulation transfer gate TGD,storage gates SG1 and SG2, FD transfer gates TG1 and TG2 and reset gatesRG1 and RG2, respectively.

A photogate voltage VPG is applied to the common photogate CPG, anoverflow gate voltage VOG is applied to the overflow gates OG1 and 0G2,storage control signals SSG1 and SSG2 are applied to the storage gatesSG1 and SG2, FD transfer control signals STG1 and STG2 are applied tothe FD transfer gates TG1 and TG2, reset signals SRG1 and SRG2 areapplied to the reset gates RG1 and RG2, and selection signals SEL1 andSEL2 are applied to the gate of the selection transistors TSL1 and TSL2.First through fourth demodulation transfer control signals STGA, STGB,STGC and STGD having different phases are applied to the first throughfourth demodulation transfer gates TGA, TGB, TGC and TGD, respectively.

The photogate voltage VPG, the overflow gate voltage VOG, the storagecontrol signals SSG1 and SSG2, the FD transfer control signals STG1 andSTG2, the reset signals SRG1 and SRG2, the selection signals SEL1 andSEL2, and the demodulation transfer control signals STGA, STGB, STGC andSTGD may be provided from the row scanning circuit 130 under control ofthe controller 150 as described above with reference to FIG. 2.

Referring to FIGS. 22 and 23, during the integration period TINT, thephotogate voltage VPG applied to the common photogate CPG may have a DCvoltage level VDC for collecting the photo charge, and the overflow gatevoltage VOG applied to the overflow gates OG1 and OG2 may have aturn-off voltage level VOFF to block draining of the photo charge. Inaddition, during the integration period TINT, the first through fourthdemodulation transfer control signals STGA, STGB, STGC and STGD ofdifferent phases may be applied to the first through fourth demodulationtransfer gates TGA, TGB, TGC and TGD, respectively. The phase of thefirst demodulation transfer control signal STGA may be synchronized withthe phase of the transmission light TL. In some example embodiments, thephase difference between the first and second demodulation transfercontrol signals STGA and STGB may be 90 degrees, the phase differencebetween the first and third demodulation transfer control signals STGAand STGC may be 180 degrees, and the phase difference between the firstand fourth demodulation transfer control signals STGA and STGD may be270 degrees. The distance measuring method using the demodulationsignals of the different phases are the same as or similar to thatdescribed above with reference to FIGS. 2 through 4.

In the example embodiment of the depth pixel PX2 of FIG. 22 having thesingle common photogate CPC, the first through fourth demodulationtransfer control signals STGA˜STGD applied to the first through fourthdemodulation transfer gates TGA˜TGD correspond to the above-describedsampling control signals. The example embodiments as described withreference to FIGS. 6 through 21 may be applied to the depth pixel PX2 ofFIG. 22 in the same way by replacing the first through fourth photogatesignals SPGA˜SPGD with the first through fourth demodulation transfercontrol signals STGA˜STGD.

FIGS. 24 and 25 are diagrams illustrating a structure of sharing afloating diffusion region of a depth pixel according to exampleembodiments.

A depth pixel PX2 of FIG. 24 is substantially the same as the depthpixel PX2 of FIG. 22 having the four-tap structure and the commonphotogate CPG

As illustrated in FIG. 24, the first floating diffusion region FDAcorresponding to the first tap TA and the second floating diffusionregion FDB corresponding to the second tap TB may be electricallyconnected to each other through a conduction path LN1. Further, thethird floating diffusion region FDC corresponding to the third tap TCand the fourth floating diffusion region FDD corresponding to the fourthtap TD may be electrically connected to each other through a conductionpath LN2. The conductions paths LN1 and LN2 may include conduction linesabove the semiconductor substrate and vertical contacts such as vias.

In case of the depth pixel PX2 of FIG. 24, a demodulation signal havinga first phase may be applied to the first and second demodulationtransfer gates TGA and TGB, and a demodulation signal having a secondphase different from the first phase may be applied to the third andfourth demodulation transfer gates TGC and TGD. As such, the sensingsensitivity of the depth pixel may be enhanced by electricallyconnecting at least two floating diffusion regions among a plurality offloating diffusion regions included in each depth pixel.

FIG. 25 illustrates four depth pixels PXa, PXb, PXc and PXD, which areadjacent in the first horizontal direction X and the second horizontaldirection Y for convenience of illustration. It is understood that morepixels may be arranged repeatedly in the pixel array 110 in FIG. 2.

Referring to FIG. 25, the four adjacent depth pixels PXa, PXb, PXc andPXD may share one floating diffusion region. For example, the seconddemodulation signal may be applied commonly to the four taps, which areadjacent to the floating diffusion region FDB and respectively includedin the four adjacent depth pixels PXa, PXb, PXc and PXD, and the photocharge collected by the four adjacent depth pixels PXa, PXb, PXc and PXDmay be summed into the centered floating diffusion region FDB. In thisway, the photo charge collected by every four adjacent depth pixels maybe summed into the floating diffusion regions FDA, FDB, FDC and FDD,respectively, depending on the phases of the demodulation signals. Assuch, the sensing sensitivity of the ToF sensor may be enhanced throughthe structure of sharing the floating diffusion region.

FIG. 26 is a block diagram illustrating an electronic device 1000according to an example embodiment, and FIG. 27 is a block diagramillustrating a camera module 1100 b included in the electronic device ofFIG. 26.

Referring to FIG. 26, an electronic device 1000 may include a cameramodule group 1100, an application processor 1200, a power managementintegrated circuit (PMIC) 1300 and an external memory 1400.

The camera module group 1100 may include a plurality of camera modules1100 a, 1100 b and 1100 c. At least one of the camera modules 1100 a,1100 b and 1100 c may include a multi-function ToF sensor as describedabove with reference to FIGS. 1 through 25. FIG. 26 illustrates threecamera modules 1100 a, 1100 b and 1100 c as an example, and it isunderstood that example embodiments are not limited to a particularnumber of camera modules. According to other example embodiments, thecamera module group 1100 may include two camera modules, and four ormore camera modules.

Hereinafter, an example configuration of the camera module 1100 b isdescribed with reference to FIG. 27. The same descriptions may beapplied to one or more of the other camera modules 1100 a and 1100 c.

Referring to FIG. 27, the camera module 1100 b may include a prism 1105,an optical path folding element (OPFE) 1110, an actuator 1130, an imagesensing device 1140 and a storage device 1150.

The prism 1105 may include a reflection surface 1107 to change a path ofa light L incident on the prism 1105.

In some example embodiment, the prism 1105 may change the path of thelight L incident in a first direction X to the path in a seconddirection Y perpendicular to the first direction X. In addition, theprism 1105 may rotate the reflection surface 1107 around a center axis1106 and/or rotate the center axis 1106 in the B direction to align thepath of the reflected light along the second direction Y. Also the OPFE1110 may move in a third direction perpendicular to the first directionX and the second direction Y.

A rotation angle of the prism 1105 may be smaller than 15 degrees in thepositive (+) A direction and greater than 15 degrees in the negative (−)A direction, but it is understood that one or more other exampleembodiments are not limited thereto.

Further, the prism 1105 may rotate within 20 degrees in the positive Bdirection and the negative B direction, but it is understood that one ormore other example embodiments are not limited thereto.

Additionally, the prism 1105 may move the reflection surface 1106 in thethird direction Z that is in parallel with the center axis 1106.

The OPFE 1110 may include optical lenses that are divided into m groupswhere m is a positive integer. The m lens group may move in the seconddirection Y to change an optical zoom ratio of the camera module 1100 b.For example, the optical zoom ratio may be changed in a range of 3K, 5K,and so on by moving the m lens group, when K is a basic optical zoomratio of the camera module 1100 b.

The actuator 1130 may move the OPFE 1110 or the optical lens to aspecific position. For example, the actuator 1130 may adjust theposition of the optical lens for accurate sensing such that an imagesensor 1142 may be located at a position corresponding to a focal lengthof the optical lens.

The image sensing device 1140 may include an image sensor 1142, acontrol logic 1144 and a memory 1146. The image sensor 1142 may captureor sense an image using the light provided through the optical lens. Thecontrol logic 1144 may control overall operations of the camera module1100 b. For example, the control logic 1144 may provide control signalsthrough a control signal line CSLb to control the operation of thecamera module 1100 b.

The memory 1146 may store information such as calibration data 1147 forthe operation of the camera module 1100 b. For example, the calibrationdata 1147 may include information for generation of image data based onthe provided light, such as information on the above-described rotationangle, a focal length, information on an optical axis, and so on. Whenthe camera module is implemented as a multi-state camera having avariable focal length depending on the position of the optical lens, thecalibration data 1147 may include multiple focal length values andauto-focusing values corresponding to the multiple states.

The storage device 1150 may store the image data sensed using the imagesensor 1142. The storage device 1150 may be disposed output of the imagesensing device 1140, and the storage device 1150 may be staked with asensor chip comprising the image sensing device 1140. The storage device1150 may be implemented with an electrically erasable programmableread-only memory (EEPROM), although it is understood that one or moreother example embodiments are not limited thereto.

Referring to FIGS. 26 and 27, each of the camera modules 1100 a, 1100 band 1100 c may include an actuator 1130. In this case, the cameramodules 1100 a, 1100 b and 1100 c may include the same or differentcalibration data 1147 depending on the operations of the actuators 1130.

In some example embodiments, one camera module 1100 b may have a foldedlens structure included the above-described prism 1105 and the OPFE1110, and the other camera modules 1100 a and 1100 b may have a verticalstructure without the prism 1105 and the OPFE 1110.

In some example embodiments, one camera module 1100 c may be a depthcamera configured to measure distance information of an object using aninfrared light. In this case, the application processor 1200 may mergethe distance information provided from the depth camera 1100 c and imagedata provided from the other camera modules 1100 a and 1100 b togenerate a three-dimensional depth image.

In some example embodiments, at least two camera modules among thecamera modules 1100 a, 1100 b and 1100 c may have different fields ofview, for example, through different optical lenses.

In some example embodiments, each of the camera modules 1100 a, 1100 band 1100 c may be separated physically from each other. In other words,the camera modules 1100 a, 1100 b and 1100 c may include a dedicatedimage sensor 1142, respectively.

The application processor 1200 may include an image processing device1210, a memory controller 1220 and an internal memory 1230. Theapplication processor 1200 may be separated from the camera modules 1100a, 1100 b and 1100 c. For example, the application processor 1200 may beimplemented as one chip and the camera modules 1100 a, 1100 b and 1100 cmay implemented as another chip or other chips.

The image processing device 1210 may include a plurality of subprocessors 1212 a, 1212 b and 1212 c, an image generator 1214 and acamera module controller 1216.

The image data generated by the camera modules 1100 a, 1100 b and 1100 cmay be provided to the sub processors 1212 a, 1212 b and 1212 c throughdistinct image signal lines ISLa, ISLb and ISLc, respectively. Forexample, the transfer of the image data may be performed using a cameraserial interface (CSI) based on mobile industry processor interface(MIPI), although it is understood that one or more other exampleembodiments are not limited thereto.

In some example embodiments, one sub processor may be assigned commonlyto two or more camera modules. In this case, a multiplexer may be usedto transfer the image data selectively from one of the camera modules tothe shared sub processor.

The image data from the sub processors 1212 a, 1212 b and 1212 c may beprovided to the image generator 1214. The image generator 1214 maygenerate an output image using the image data from the sub processors1212 a, 1212 b and 1212 c according to image generating information or amode signal. For example, the image generator 1213 may merge at least aportion of the image data from the camera modules 1100 a, 1100 b and1100 c having the different fields of view to generate the output imageaccording to the image generating information or the mode signal. Inaddition, the image generator 1214 may select, as the output image, oneof the image data from the camera modules 1100 a, 1100 b and 1100 caccording to the image generating information or the mode signal.

In some example embodiments, the image generating information mayinclude a zoom factor or a zoom signal. Further, the mode signal may bea signal based on a selection of a user.

When the image generating information is the zoom factor and the cameramodules 1100 a, 1100 b and 1100 c have the different fields of view, theimage generator 1214 may perform different operations depending on thezoom signal. For example, when (or based on) the zoom signal is a firstsignal, the image generator 1214 may merge the image data from thedifferent camera modules to generate the output image. When the zoomsignal is a second signal different from the first signal, the imagegenerator 1214 may select, as the output image, one of image data fromthe camera modules 1100 a, 1100 b and 1100 c.

In some example embodiments, the image generator 1214 may receive theimage data of different exposure times from the camera modules 1100 a,1100 b and 1100 c. In this case, the image generator 1214 may performhigh dynamic range (HDR) processing with respect to the image data fromthe camera modules 1100 a, 1100 b and 1100 c to generate the outputimage having the increased dynamic range.

The camera module controller 1216 may provide control signals to thecamera modules 1100 a, 1100 b and 1100 c. The control signals generatedby the camera module controller 1216 may be provided to the cameramodules 1100 a, 1100 b and 1100 c through the distinct control signallines CSLa, CSLb and CSLc, respectively.

In some example embodiments, one of the camera modules 1100 a, 1100 band 1100 c may be designated as a master camera according to the imagegenerating information of the mode signal, and the other camera modulesmay be designated as slave cameras.

The camera module acting as the master camera may be changed accordingto the zoom factor or an operation mode signal. For example, when thecamera module 1100 a (e.g., first camera module0 has the wider field ofview than the camera module 1100 b (e.g., second camera module) and thezoom factor indicates a lower zoom magnification, the camera module 1100b may be designated as the master camera. In contrast, when the zoomfactor indicates a higher zoom magnification, the camera module 1100 amay be designated as the master camera.

In some example embodiments, the control signals provided from thecamera module controller 1216 may include a synch enable signal. Forexample, when the camera module 1100 b is the master camera and thecamera modules 1100 a and 1100 c are the slave cameras, the cameramodule controller 1216 may provide the synch enable signal to the cameramodule 1100 b. The camera module 1100 b may generate a synch signalbased on the provided synch enable signal and provide the synch signalto the camera modules 1100 a and 1100 c through a synch signal line SSL.As such, the camera modules 1100 a, 1100 b and 1100 c may transfer thesynchronized image data to the application processor 1200 based on thesynch signal.

In some example embodiments, the control signals provided from thecamera module controller 1216 may include information on the operationmode. The camera modules 1100 a, 1100 b and 1100 c may operate in afirst operation mode or a second operation mode based on the informationfrom the camera module controller 1216.

In the first operation mode, the camera modules 1100 a, 1100 b and 1100c may generate image signals with a first speed (e.g., a first framerate) and encode the image signals with a second speed higher than thefirst speed (e.g., a second frame rate higher than the first frame rate)to transfer the encoded image signals to the application processor 1200.The second speed may be less than thirty times the first speed. Theapplication processor 1200 may store the encoded image signals in theinternal memory 1230 or the external memory 1400. The applicationprocessor 1200 may read out and decode the encoded image signals toprovide display data to an display device. For example, the subprocessors 1212 a, 1212 b and 1212 c may perform the decoding operationand the image generator 1214 may process the decoded image signals.

In the second operation mode, the camera modules 1100 a, 1100 b and 1100c may generate image signals with a third speed lower than the firstspeed (e.g., the third frame rate lower than the first frame rate) totransfer the generated image signals to the application processor 1200.That is, the image signals that are not encoded may be provided to theapplication processor 1200. The application processor 1200 may processthe received image signals or store the receive image signals in theinternal memory 1230 or the external memory 1400.

The PMIC 1300 may provide a power supply voltage to the camera modules1100 a, 1100 b and 1100 c, respectively. For example, the PMIC 1300 mayprovide, under control of the application processor 1200, a first powerto the camera module 1100 a through a power line PSLa, a second power tothe camera module 1100 b through a power line PSLb, and a third power tothe camera module 1100 c through a power line PSLc.

The PMIC 1300 may generate the power respectively corresponding to thecamera modules 1100 a, 1100 b and 1100 c and control power levels, inresponse to a power control signal PCON from the application processor1200. The power control signal PCON may include information on the powerdepending on the operation modes of the camera modules 1100 a, 1100 band 1100 c. For example, the operation modes may include a low powermode in which the camera modules 1100 a, 1100 b and 1100 c operate withlow powers. The power levels of the camera modules 1100 a, 1100 b and1100 c may be the same as or different from each other. In addition, thepower levels may be changed dynamically or adaptively.

The inventive concept may be applied to any electronic devices andsystems including a ToF sensor. For example, the inventive concept maybe applied to systems such as a mobile phone, a smart phone, a personaldigital assistant (PDA), a portable multimedia player (PMP), a digitalcamera, a camcorder, a personal computer (PC), a server computer, aworkstation, a laptop computer, a digital TV, a set-top box, a portablegame console, a navigation system, a wearable device, an internet ofthings (IoT) device, an internet of everything (IoE) device, an e-book,a virtual reality (VR) device, an augmented reality (AR) device, avehicle navigation device, a video phone, a monitoring system, an autofocusing system, a tracking system, a motion detection system, etc.

The foregoing is illustrative of example embodiments and is not to beconstrued as limiting thereof. Although a few example embodiments havebeen described, those skilled in the art will readily appreciate thatmany modifications are possible in the example embodiments withoutmaterially departing from the present inventive concept.

1. A method of operating a time-of-flight (ToF) sensor including at least one depth pixel having a multi-tap structure and a light source illuminating a transmission light to an object, the method comprising: determining an operation mode of the ToF sensor, from among a distance detection mode to sense a distance to an object and a plurality of additional operation modes; controlling a plurality of taps of a depth pixel, among the at least one depth pixel, and the light source based on the determined operation mode such that the plurality of taps generate a plurality of sample data corresponding to the determined operation mode; and determining a sensing result corresponding to the determined operation mode based on the plurality of sample data.
 2. The method of claim 1, wherein the plurality of additional operation modes includes an object detection mode to sense existence of the object, a motion detection mode to sense a motion of the object, a combination detection mode to simultaneously sense the existence of the object and the motion of the object, and a wide dynamic range (WDR) mode to sense the object with a plurality of sensing sensitivities.
 3. The method of claim 1, further comprising: based on the determined operation mode being one of the plurality of additional operation modes, dividing an integration period to collect a photo charge generated by an incident light into a plurality of shot periods; and selectively activating the transmission light and a plurality of sampling control signals to control the plurality of taps during the plurality of shot periods based on the determined operation mode.
 4. The method of claim 1, wherein at least one of a plurality of sampling control signals to control the plurality of taps is deactivated during an integration period to collect a photo charge generated by an incident light, to generate noise sample data indicating a sensing noise of the depth pixel.
 5. The method of claim 1, wherein the controlling the plurality of taps and the light source comprises: based on the determined operation mode being the distance detection mode, applying a plurality of sampling control signals of different phases to the plurality of taps during an integration period to collect a photo charge generated by an incident light; and based on the determined operation mode being one of the plurality of additional operation modes, dividing the integration period into a plurality of shot periods, selectively activating the plurality of sampling control signals during the plurality of shot periods based on the determined operation mode and applying the plurality of sampling control signals to the plurality of taps.
 6. The method of claim 1, wherein the controlling the plurality of taps and the light source comprises: based on the determined operation mode being the distance detection mode, controlling the light source to generate the transmission light modulated with a modulation frequency during an integration period to collect a photo charge generated by an incident light; and based on the determined operation mode being one of the plurality of additional operation modes, dividing the integration period into a plurality of shot periods and controlling the light source to generate the transmission light that is selectively activated during the plurality of shot periods based on the determined operation mode.
 7. The method of claim 1, further comprising: changing the determined operation mode based on the sensing result.
 8. The method of claim 1, wherein a number of the plurality of taps is greater than two.
 9. The method of claim 1, wherein the controlling the plurality of taps and the light source comprises, based on the determined operation mode being an object detection mode to sense existence of the object: dividing an integration period to collect a photo charge generated by an incident light into a first shot period and a second shot period; deactivating the transmission light during the first shot period; generating ambient light sample data corresponding to an ambient light by activating at least one first signal among a plurality of sampling control signals to control the plurality of taps during the first shot period; activating the transmission light during the second shot period; and generating object sample data corresponding to the object by activating at least one second signal among the plurality of sampling control signals during the second shot period.
 10. The method of claim 9, wherein the determining the sensing result comprises, based on the determined operation mode being the object detection mode: determining whether the object exists within a reference distance based on a value of the object sample data subtracted by the ambient light sample data.
 11. The method of claim 1, wherein the controlling the plurality of taps and the light source comprises, based on the determined operation mode being a motion detection mode to sense a motion of the object: dividing an integration period to collect a photo charge generated by an incident light into a first shot period and a second shot period; generating first object sample data corresponding to the object by activating at least one first signal among a plurality of sampling control signals to control the plurality of taps during the first shot period; and generating second object sample data corresponding to the object by activating at least one second signal among the plurality of sampling control signals during the second shot period.
 12. The method of claim 11, wherein the determining the sensing result comprises, based on the determined operation mode being the motion detection mode: determining the motion of the object based a difference value between the first object sample data and the second object sample data.
 13. The method of claim 1, wherein the controlling the plurality of taps and the light source comprises, based on the determined operation mode being a motion detection mode to sense a motion of the object: dividing an integration period to collect a photo charge generated by an incident light into a plurality of shot periods; and generating a plurality of object sample data corresponding to the object by sequentially activating a plurality of sampling control signals to control the plurality of taps during the plurality of shot periods.
 14. The method of claim 1, wherein the controlling the plurality of taps and the light source comprises, based on the determined operation mode being a combination detection mode to sense both an existence of the object and a motion of the object: dividing an integration period to collect a photo charge generated by an incident light into a first shot period, a second shot period and a third shot period; deactivating the transmission light during the first shot period; generating ambient light sample data corresponding to an ambient light by activating at least one first signal among a plurality of sampling control signals to control the plurality of taps during the first shot period; activating the transmission light during the second shot period and the third shot period; generating first object sample data corresponding to the object by activating at least one second signal among the plurality of sampling control signals during the second shot period; and generating second object sample data corresponding to the object by activating at least one third signal among the plurality of sampling control signals during the third shot period.
 15. The method of claim 14, wherein the determining the sensing result comprises, based on the determined operation mode being the combination detection mode: determining whether the object exists within a reference distance based on a value of the first object sample data subtracted by the ambient light sample data and the second object sample data subtracted by the ambient light sample data; and determining the motion of the object based a difference value between the first object sample data and the second object sample data.
 16. The method of claim 1, wherein the controlling the plurality of taps and the light source comprises, based on the determined operation mode being a WDR mode to sense the object with a plurality of sensing sensitivities: dividing an integration period to collect a photo charge generated by an incident light into a first shot period, a second shot period longer than the first shot period and a third shot period longer than the second shot period; generating first sensitivity sample data by activating at least one first signal among a plurality of sampling control signals to control the plurality of taps during the first shot period; generating second sensitivity sample data by activating at least one second signal among the plurality of sampling control signals during the second shot period; and generating third sensitivity sample data by activating at least one third signal among the plurality of sampling control signals during the third shot period.
 17. A time-of-flight sensor comprising: a light source configured to illuminate a transmission light to an object; a pixel array comprising a depth pixel having a multi-tap structure; a row scanning circuit configured to generate a plurality of sampling control signals applied to a plurality of taps of the depth pixel; and a controller configured to control the light source, the pixel array and the row scanning circuit based on a mode signal indicating a selected operation mode of a ToF sensor, selected from among a distance detection mode to sense a distance to an object and a plurality of additional operation modes.
 18. The ToF sensor of claim 17, wherein the row scanning circuit is further configured to: based on the selected operation mode being the distance detection mode, apply a plurality of sampling control signals of different phases to the plurality of taps during an integration period to collect a photo charge generated by an incident light; and based on the selected operation mode being one of the plurality of additional operation modes, divide the integration period into a plurality of shot periods, selectively activate the plurality of sampling control signals during the plurality of shot periods based on the selected operation mode and apply the plurality of sampling control signals to the plurality of taps.
 19. The ToF sensor of claim 17, wherein the light source is further configured to: based on the selected operation mode being the distance detection mode, generate the transmission light modulated with a modulation frequency during an integration period to collect a photo charge generated by an incident light; and based on the selected operation mode being one of the plurality of additional operation modes, divide the integration period into a plurality of shot periods and generate the transmission light that is selectively activated during the plurality of shot periods based on the selected operation mode.
 20. A method of operating a time-of-flight (ToF) sensor including at least one depth pixel having a multi-tap structure and a light source illuminating a transmission light to an object, the method comprising: determining an operation mode of a ToF sensor, from among a distance detection mode to sense a distance to an object and a plurality of additional operation modes; based on the determined operation mode being the distance detection mode, applying a plurality of sampling control signals of different phases to a plurality of taps of a depth pixel during an integration period to collect a photo charge generated by an incident light; and based on the determined operation mode being one of the plurality of additional operation modes, dividing the integration period into a plurality of shot periods and selectively activating the transmission light and the plurality of sampling control signals during the plurality of shot periods based on the determined operation mode. 21-30. (canceled) 